Skip to content

Add OSV-Scanner security gate + clear all CVEs (Python floor to 3.10)#798

Open
vikrantpuppala wants to merge 6 commits into
mainfrom
vp/security-scan
Open

Add OSV-Scanner security gate + clear all CVEs (Python floor to 3.10)#798
vikrantpuppala wants to merge 6 commits into
mainfrom
vp/security-scan

Conversation

@vikrantpuppala

@vikrantpuppala vikrantpuppala commented May 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

This PR adds the OSV-Scanner security gate and clears all open dependency CVEs on the connector — without forcing any dependency-pin changes on consumers. The only breaking change is raising the Python floor to 3.10, which is required because the CVE-fixed cryptography and pyjwt releases are not installable on 3.8/3.9.

What it does

1. OSV-Scanner security workflow (securityScan.yml), hardened fail-closed

  • Single workflow, three triggers: PR (fail on CVSS ≥ 7 or any unscored finding), weekly cron, and manual — reading poetry.lock natively.
  • Ported the fail-closed hardening from the Go (databricks/databricks-sql-go#362) and Node (databricks/databricks-sql-nodejs#388) driver gates:
    • Capture osv-scanner's exit code; tolerate only 0/1 and fail closed on any other code (network error, corrupt binary) instead of masking with || true.
    • Validate the output is well-formed JSON with a .results array before parsing (partial/corrupt scan → fail closed, not "0 findings").
    • Resolve empty group max_severity via a cvss_num fallback to an UNKNOWN sentinel (using try (x|tonumber) catch null, not tonumber?) so a scoreless finding can never sort to 0 and sail past the gate. UNKNOWN always blocks.
    • Integer-count guards fail closed on parse failure.
    • Per-repo SMTP email dropped in favor of artifact upload for the planned cross-repo collator (parity with Go/Node).

2. Clears all CVEs by refreshing poetry.lock — no forced pin changes

  • Every runtime dependency pin is unchanged (thrift ~=0.22.0, urllib3 >=1.26, requests ^2.18.1, pyjwt ^2.0.0, pyarrow floors). The existing constraints already allow the CVE-free versions; the refreshed lock simply resolves to them: urllib3 2.7.0, cryptography 49.0.0, pyarrow 23.0.1, requests 2.34.2, pyjwt 2.13.0, idna 3.18, python-dotenv 1.2.2.
  • Consumers do not need us to relax or raise any pin to become CVE-free — they only need Python ≥ 3.10 (their environment, not our constraint).

3. Raises the Python floor to ^3.10 (the only breaking change)

  • The fixed cryptography (≥46) and pyjwt (≥2.12) require Python ≥3.10 upstream, so a single CVE-clean lockfile cannot span 3.8/3.9. Both are long past EOL (3.8 EOL 2024-10, 3.9 EOL 2025-10) and no supported DBR LTS is affected. CI matrices updated to drop 3.9; Requires-Python fails safe for anyone still on 3.9 (they stay on 4.3.x).

4. thrift stays ~=0.22.0, documented accurately

  • thrift 0.22.0 is affected by open Apache Thrift advisories (CVE-2025-48431 + the CVE-2026-41602..41636 set), but all are in non-Python bindings (Node.js / Go / c_glib / Java / Swift — verified against the upstream advisories), so the Python lib/py code we ship is not affected. The only fix (0.23.0) is the version that caused the SEV0 ES-1960554 DBR-LTS install break, so we cannot take it until a build-safe thrift ships (THRIFT-6067).
  • These advisories are invisible to OSV-Scanner and Dependabot (both match by PyPI purl, but these CVEs are filed only against the upstream CPE cpe:2.3:a:apache:thrift with no package coordinate). A supplementary weekly NVD-CPE thrift watch in the workflow surfaces them and hard-fails if a Python-affecting thrift CVE ever appears. Scoped to thrift alone because an audit of all three drivers' full dependency sets found thrift is the only dep with this purl-vs-CPE gap.

5. Dev-tool + test fixes surfaced by the refresh

  • Bump dev-only black ^22 → ^26 and pytest ^7 → ^9 to clear their advisories (never shipped in the wheel); reformat src with black 26.
  • Fix 2 mypy errors surfaced by the newer mypy (auth/oauth.py, auth/retry.py) — type-annotation-only.
  • Declare pytz as an explicit dev dependency (pandas 3.0 dropped it as a required dep; test_parameters.py imports it directly).

Result

  • OSV-Scanner v2.3.8: 0 findings on the refreshed lock; osv-scanner.toml needs no suppressions.
  • black --check passes; mypy passes; the Python matrix (3.10–3.14) is green.

Test plan

  • OSV-Scanner v2.3.8 run locally against the refreshed poetry.lock → 0 findings, 0 blocking.
  • Hardening verified against synthetic OSV JSON (scanner-error, partial-JSON, empty-severity, scoreless-UNKNOWN all fail closed).
  • black --check src passes with black 26.5.1.
  • mypy src passes (2 pre-existing latent errors fixed).
  • Thrift NVD-CPE watch tested against live NVD (finds the CPE-only advisories, correctly flags 0 as Python-affecting).
  • Verified all 3 drivers audited for the purl-vs-CPE gap: thrift is the only affected dep; Go/Node already ship the fixed thrift 0.23.0.

This pull request and its description were written by Isaac.

@vikrantpuppala vikrantpuppala mentioned this pull request Jun 1, 2026
Open
5 tasks
@vikrantpuppala
Add OSV-Scanner-based security workflow
ec9f1ef 
Single workflow, single job, three triggers:
  - pull_request to main: fails on CVSS >= 7 findings only
    (HIGH/CRITICAL block merges; MED/LOW visible but non-blocking)
  - cron weekly (Sunday 00:00 UTC): reports ALL findings via email
  - workflow_dispatch: behaves like cron

Mirrors the JDBC driver's security workflow (databricks-jdbc#1460)
adapted for Python:
  - Reads poetry.lock natively via OSV-Scanner --lockfile (no
    separate SBOM tool needed)
  - Reuses the existing ./.github/actions/setup-jfrog composite action
    for parity with other workflows (the workflow functionally doesn't
    need JFrog since OSV reads the lockfile directly, but keeping the
    composite action preserves the established pattern)
  - Suppressions in osv-scanner.toml ([[IgnoredVulns]] schema)

The workflow is not yet wired into branch protection. Day-one scan
against current main surfaces 14 HIGH / 10 MED / 1 LOW (25 total) --
concentrated in cryptography, urllib3, pyjwt, pyarrow, requests,
black, pytest, python-dotenv, idna. These will be addressed by a
follow-up dep-bump PR.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
@vikrantpuppala
Harden OSV gate (fail-closed) + refresh lockfile to clear all CVEs
02f2f6b 
Port the fail-closed hardening from the Go (#362) and Node (#388) OSV
workflows, and refresh poetry.lock so the gate passes with zero
suppressions.

securityScan.yml hardening (was fail-open in three places):
- Capture osv-scanner's exit code; tolerate only 0/1 and fail closed on
  any other code (network error, corrupt binary) instead of masking it
  with `|| true`.
- Validate the output is well-formed JSON with a .results array before
  parsing, so a truncated/partial scan fails closed rather than parsing
  to zero findings.
- Resolve empty group max_severity via a cvss_num fallback to an
  UNKNOWN sentinel (using `try (x|tonumber) catch null`, not
  `tonumber?`), so a scoreless finding can never sort to 0 and sail past
  the CVSS>=7 gate. UNKNOWN always blocks (PyPA advisories carry CVSS; a
  scoreless finding is a GHSA-only/malware advisory).
- Integer-count guards fail closed on parse failure.
- Drop per-repo SMTP email in favor of artifact upload for the planned
  cross-repo collator (parity with Go/Node).

CVE clearing WITHOUT forcing dependency floors:
- Bump the Python floor to ^3.10. The CVE-fixed cryptography (>=46) and
  pyjwt (>=2.12) require Python >=3.10 upstream, so a single CVE-clean
  lockfile cannot span 3.8/3.9. This is the only breaking change.
- All runtime dependency pins are UNCHANGED (thrift ~=0.22.0,
  urllib3 >=1.26, requests ^2.18.1, pyjwt ^2.0.0, pyarrow floors). The
  existing constraints already ALLOW the CVE-free versions; the refreshed
  lock simply resolves to them (urllib3 2.7.0, cryptography 49.0.0,
  pyarrow 23.0.1, requests 2.34.2, pyjwt 2.13.0, idna 3.18,
  python-dotenv 1.2.2). Customers do not need us to relax or raise any
  pin to become CVE-free.
- thrift stays ~=0.22.0 (no known advisory; the <0.23 cap avoids the
  ES-1960554 DBR-LTS install break).
- Bump dev-only black ^22 -> ^26 and pytest ^7 -> ^9 to clear their
  advisories (never shipped in the wheel); reformat src with black 26.

Result: OSV-Scanner v2.3.8 reports 0 findings on the refreshed lock;
osv-scanner.toml needs no suppressions.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
@vikrantpuppala
ci: drop Python 3.9 from CI matrices to match the ^3.10 floor
334331a 
The pyproject floor is now ^3.10, so 3.9 legs can no longer `poetry
install` (^3.10 is unsatisfiable on a 3.9 interpreter) and would fail.
Remove "3.9" from every unit-test / lint / type-check / pyarrow / kernel
matrix in code-quality-checks.yml and warm-deps-cache.yml, and drop the
now-moot 3.9-kernel exclude in the warm-deps cache.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
@vikrantpuppala
Document thrift CPE-gap accurately + add weekly NVD-CPE thrift watch
30aa6e3 
Corrects the earlier "thrift ~=0.22.0 (no known advisory)" wording, which
was wrong: thrift 0.22.0 IS affected by open Apache Thrift advisories
(CVE-2025-48431 + the CVE-2026-41602..41636 set, all fixed in 0.23.0).

Why we still hold at ~=0.22.0 and accept them:
- Apache Thrift is one monorepo shipping ~20 language libraries; the PyPI
  `thrift` package is built only from lib/py. Each of these CVEs is in a
  NON-Python binding -- verified against the upstream oss-security
  advisories: Node.js (41636), Go (41602), c_glib/C (48431), Java (41603),
  Swift (41604, 41605). None touches the Python code paths we ship.
- The only fix (0.23.0) is the version that caused SEV0 ES-1960554 on
  DBR-LTS old setuptools, so we cannot take it until a build-safe thrift
  ships (THRIFT-6067).

Why the OSV gate doesn't flag it (and why that is NOT proof Python is safe):
- These CVEs are in OSV with `affected[].package = null` -- only a GIT/CPE
  coordinate (cpe:2.3:a:apache:thrift), no PyPI/npm/Go package entry. OSV
  and Dependabot both match by package purl, so they return nothing for
  PyPI thrift. This is a coordinate blind spot, independent of whether
  Python is affected -- a FUTURE Python-affecting thrift CVE filed the same
  way would also be missed.

Mitigation: a supplementary NVD-CPE thrift watch in securityScan.yml,
scheduled/manual only (never PR; NVD rate limits). It lists all
apache:thrift CVEs affecting the locked version in the weekly summary and
hard-fails if any description names Python. Scoped to thrift alone because
an audit of all three drivers' full dependency sets found thrift is the
only dep with this purl-vs-CPE gap (Go/Node already ship the fixed 0.23.0).
The Python-detection is a heuristic (description must say python/lib/py);
the full list is always surfaced for human review.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
@vikrantpuppala
fix(types): resolve 2 mypy errors surfaced by the refreshed mypy
bb61bd8 
The lockfile refresh floats mypy (^1.10.1) up to 1.20.2, which is stricter
and flags two pre-existing latent type issues that the older mypy missed:

- auth/oauth.py:48 — is_expired() returned `exp_time and (...)`, whose value
  is `Any | None` (the exp claim) when falsy, not bool, violating the
  `-> bool` annotation. Use `exp_time is not None and (...)` so the return
  is a real bool and the None-exp case is explicit.
- auth/retry.py:248 — the command_type setter was annotated
  `value: CommandType`, but the getter returns `Optional[CommandType]` and
  __private_init__ assigns an `Optional[CommandType]`. Widen the setter to
  `Optional[CommandType]` to match the getter and actual usage.

Both are type-annotation-only changes; no runtime behavior change.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
@vikrantpuppala
fix(tests): declare pytz explicitly (pandas 3.0 dropped it as a hard …
f0dae00 
…dep)

tests/unit/test_parameters.py imports pytz directly. It previously arrived
transitively via pandas, but the refreshed lock resolves pandas 3.0.3 on
Python >=3.11, and pandas 3.0 removed pytz from its required dependencies
(it is now only a pandas extra). That broke test collection on 3.11+ with
`ModuleNotFoundError: No module named 'pytz'`.

Add pytz as an explicit dev dependency so the test suite no longer relies on
pandas's transitive graph, which differs across the Python matrix.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
@vikrantpuppala vikrantpuppala changed the title Add OSV-Scanner-based security workflow Add OSV-Scanner security gate + clear all CVEs (Python floor to 3.10) Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vikrantpuppala